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IMAGE PROCESSING METHOD AND DEVICE, IMAGE PROCESSING PANEL, AND 
RECORDING MEDIUM 



(57) In accordance with calibration data collected in 
a 1st state in which a 1st camera assembled with a 2nd 
camera and a 3rd camera are unified to construct a ster- 
eocamera and calibration data collected in a 2nd state 
in which the 1st camera is disassembled from the 2nd 
and 3rd cameras and the three cameras are assembled 
again to construct a stereocamera, information on the 
rotary conversion of the stereocamera and information 
on the translation conversion thereof are calculated 
and. in accordance with the calculated information, the 
calibration data on the stereocamera in the 2nd state is 
calculated. 
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Description 

Technical Field 

5 (0001 ] This invention relates to a picture processing method and a picture processing apparatus adapted for removing 
unnecessary portion from original picture to synthesize it with other picture, a picture processing panel tor synthesizing 
character or graphic with picture imaged by image pick-up unit in the state where positions on three-dimension are 
caused to be in correspondence with each other to output it. and a recording medium on which processing procedure 
for executing the picture processing is recorded. 

10 

Background Art 

[0002] There has been conventionally known stereo camera system as a three-dimensional position information 
detecting unit in which two video cameras (hereinafter simply referred to as camera as occasion may demand) or more 
is are used to constitute so-called stereo camera to pick up image of object within three-dimensional space by the respec- 
tive cameras to obtain position information within the three-dimensional space of the object from two-dimensional pic- 
ture obtained as the result of the image pick-up. 

[0003] The detail of the stereo camera system is disclosed in. e.g.. Koichiro Deguchi "Geometry for computer vision: 
2. Explanation of mechanism of stereo" Information Processing. Vol. 37. no. 7. pp. 662-670. July. 1996. etc. 
20 [0004] In the stereo camera system, it is possible to specify, from, position information of object projected onto light 
receiving surface (hereinafter referred to as screen as occasion may demand) of the photo-electric conversion element, 
e.g.. CCD. etc. in respective cameras when image of object from which attempt is made to obtain position information 
of three-dimensional space is picked up by plural cameras, position information within the three-dimensional space of 
that object. Accordingly, it is necessary to determine in advance correspondence relationship (position information cor- 
ps respond ence relationship) between position information of object existing at a certain position within the three-dimen- 
sional space and position information at the position where its object is projected onto screen of the respective cameras 
when the object is located at that position. This operation for determining position information correspondence relation- 
ship is called calibration. Such position information correspondence relationship is determined by the calibration unit. 
Outline of the calibration unit is shown in FtG. 1. 
30 [0005] In FIG. 1 . pipes 1 01 and 1 02 are included in the same plane within three-dimensional space, and are disposed 
in such a manner m parallel with each other. A carriage 103 is installed so that it can be smoothly moved along these 
pipes 101 and 102. In addition, cameras 104 and 105 constituting stereo camera are attached at the carriage 103. 
[0006] Scale is provided at the pipes 101 and 102 or any one of them, and quantity where the carriage 103 is slid can 
be measured. A flat plate 106 on which square lattice pattern is described is installed perpendicularly to direction in 
35 which the carriage 103 is slid. As shown in FIG. 1 . the lateral direction of square lattice is assumed to be x-axis, the 
longitudinal direction is assumed to be y-axis, and the sliding direction, i.e., the direction perpendicular to the square 
lattice is assumed to be z-axis. The area of the camera side relative to this flat plate 106 is indicated by z > O. Such 
three dimensional coordinate system composed of x-axis. y-axis and z-axis is defined as world coordinate system. 
[0007] Measurement of calixation is carried out by shifting position of the carriage 103 on which the cameras 104 
40 and 105 are mounted to pick up image ai the flat plate 106 from two portions. FIG. 2 is an explanatory view in the case 
where image of object is taken from two portions, and is a view when the unit of FIG. 1 is viewed from just above. 
[0008] Initially, the cameras 104 and 105 are fixed at a certain position P, to pick up image of flat plate 106 so that 
square lattice is imaged thereafter to move the cameras 104 and 105 by distance Lto other position P 2 by slide of the 
carriage 103 to pick up image of the flat plate 106 for a second time, tn this example, while the cameras 104 and 105 
45 are slid in a direction away from the flat plate 106 in this FIG. 2. its slide direction may be opposite to each other. 

[0009] Moreover, two-dimensional picture obtained by sliding the cameras 1 04 and 1 05 in this way to pick up image 
of the flat plate 106 can be also obtained by sliding the flat plate 106 in the state where the cameras 104 and 105 are 
fixed as shown in FIG. 3. 

[0010] Namely, even if there is also employed an approach as shown in FIG. 3 to fix cameras 104 and 105 at a certain 
» position P t to pick up image of the flat plate 106 so that its square lattice can be imaged thereafter to slide the flat plate 

106 by distance L in a direction away from the cameras 104 and 105 along the z-axis of the flat plate 106 to pick up 

image of the flat plate 106 at that position for a second time, simitar two-dimensional picture can be obtained. 

[001 1 ] In this FIG. 3. when left and lower corner of square lattice (first square lattice Q,) described on the flat plate 

106 before the fiat plate 106 is away by distance L is caused to be origin and is set as origin of the world coordinate. 
55 position of (i. 0 is set to (i. j. O) in the world coordinates in the fast square lattice of this Rat plate 106. In addition. 

position of ft. j) in the square lattice (second square lattice Q*) of the flat plate 106 after slid by length L is (i. j.-L) in the 

world coordinates. 

[0012] FIG. 4 is a view showing first square lattice Q, and second square lattice on the camera 104 and the flat 



2 



EP0923 235A1 



plate 106. The optical center of the camera 104 is O,. and position information of object is projected on the screen serv- 
ing as light receiving surface of. e.g.. CCD 1 1 1. etc. For example, coordinate position (p. q) in the first square lattice 
is assumed to be imaged at coordinate position (h. k) on the CCD ill. and coordinate position (r. s) in the second 
square lattice Q 2 is assumed to be imaged thereat. In addition, coordinates of portions except for cross points of 
5 respective longitudinal lines and lateral lines of respective square lattices can be determined by interpolation. 

[0013] When this is explained for a second time by using world coordinates, coordinate positions (p. q. O) and (r. s.- 
L) on the three dimension are imaged at the coordinate position (h. k) on the CCD ill. Namely, when straight line con- 
necting coordinate position (h. k) on the two dimension and coordinate positions (p. q. O) and (r. s.-L) on the three 
dimension is assumed to be 1 . it is seen that points on this line 1 are all imaged at coordinate position (h. k) on the CCD 

w 111. 

[0014] Accordingly, the line 1 represents correspondence relationship (position information correspondence relation- 
ship) between position information of object in the three-dimensional space (coordinates in the work! coordinate system 
here) and position information of two-dimensional picture obtained by picking up image of that object (coordinates in 
the two-dimensional coordinate system on the CCD 1 1 1 here). 
is [0015] This line 1 can be obtained as (x-r)/(p-r) = (y-s)/(q-s) = (z+L)/L 

[001 6] In a manner to determine the line 1 . also with respect to other coordinate positions in the two-dimensional coor- 
dinate system on the CCD 1 1 1 . lines as sets of points on the three-dimensional space projected thereonto are deter- 
mined. Further, similar fact is carried out also with respect to the camera 105. 

[001 7] As stated above, linear lines with respect to cameras 104 and 105 are all determined to thereby complete cal- 
20 ibration of the stereo camera system. 

[0018] In the stereo camera system in which calibration is carried out in this way. position information of object within 
three-dimensional space can be determined in a manner as described below. 

[0019] Initially, image of object is taken by stereo camera. Thus, as shown in FIG. 5. for example, object is assumed 
to be imaged at coordinate position (a. b) of screen (CCD) 1 11 of the camera 104. Since line 1 , in the world coordinate 
25 system corresponding to the coordinate position (a. b) is determined in the calibration, object exists on this line 1 1 in the 
world coordinate system. 

[0020] Moreover, lines in the world coordinate system corresponding to respective positions of the screen (CCD) 1 1 2 
of the camera 105 are also determined in advance in the calibration. Accordingly, it is possible to select lines having 
points crossing with the line 1 , of these lines. Lines thus selected are 1 1 3, , 1 1 3 2 . 1 1 3j • * * in the figure. 
30 [0021] Point on the screen 1 1 2 of the camera 105 corresponding to line 113, is 114! . Similarly, point corresponding 
to 113a is 114 2 and P° irTt corresponding to 1l3j is 114j. Other points are similar to the above. Accordingly, when all 
these points 1 14., . 1 14^ 1 14; • • • are gathered, there results line called 1 15 of FIG. 5. This line 1 15 is generally called 
Epipolar Line. 

[0022] As described above, object imaged at coordinate position (a, b) of screen n 1 of the camera 104 exists on any 
35 portion of the line 1 Accordingly, that object should have been projected on any portion of the Epipolar Line 1 15 on 
the screen 1 12 of the camera 105. Accordingly, it is sufficient to search projected image of object on the Epipolar line 
1 15. For example, if that projected image exists on the point (c. d) on the Epipolar line 1 1 5. line 1 2 corresponding to the 

point (c, d) is discriminated. This line 1 2 is one of lines 1 13,. 1 132. 1 13 i • Late 1 ", by determining cross point of line 

1 ! and line 1 2 by the world coordinate system, position 1 16 in the three-dimension of object can be known. Namely, it 
40 is possfole to determine three-dimensional position of object. 

[0023] At this time, the three-dimensional coordinate system serving as reference is the coordinate system peculiar 
to position and direction of the stereo camera. H the stereo camera is not moved, the three-dimensional coordinates 
serving as reference are stationary at all times. However, when the stereo camera is moved during image pick-up. the 
three-dimensional coordinates are also moved. In this example, the camera 104 and the camera 105 are represented 
45 as stereo camera in the following description. Namely, even if the expression "stereo camera" is simply employed, this 
camera system comprise, e.g.. two cameras. 

[0024] For example, as shown in FIG. 6. position of stationary object 121 determined from stereo camera 120 at initial 
time is assumed to be (x. y. z) when viewed from coordinate system 123 with the stereo camera 120 at the initial time 
being as reference. Thereafter, the stereo camera 120 is caused to undergo rotational transform R1 and transfational 
so transform T1 . and has been stereo camera 124 at the next time. 

[0025] Thus, position of the stationary object 121 determined by information from the stereo camera 1 24 at the next 
tune is expressed as (x. y, z) * R1 + T1 . 

[0026] The reason thereof is as follows. Since the relationship that the rotational transform R1 and the translational 
transform T1 are implemented holds in the coordinate system 123 and the coordinate system 125. when position of the 
65 same object 121 is measured, it is a matter of course that its positional relationship is the relationship in which the rota- 
tional transform R1 and the translational transform T1 are implemented. 

[0027] Meanwhile, in recent years, in the television station, such an approach has been carried out to pick up picture 
images at the studio (set) called virtual studio or virtual set to synthesize the picture prepared by CG (Computer 
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Graphic) with the imaged picture to broadcast this synthetic picture. 

[0028] For example, when an example of the news broadcast is taken, performer is first caused to stand with the back- 
ground of blue being as background. Image of this performer is picked up (taken) by the stereo camera. One of these 
stereo cameras is called main camera. CG is synthesized with picture (called original picture) from this main camera to 

s prepare final synthetic picture. 

[0023] Marker (pattern) which is blue system and is not conspicuous as indicated by reference numeral 126 in FIG. 
7 is attached at the background of blue. Position on three-dimension of the marker 1 26 is measured in advance with the 
world coordinate system 127 which is a specific absolutely stationary coordinate system being as reference. The 
marker 1 26 is not conspicuous, but can be barely recognized by the stereo camera. Accordingly, position of the marker 

w 126 can be measured. Namely, three-dimensional position of the marker 126 when viewed from the coordinate system 
129 with the stereo camera 128 at that time being as reference can be recognized (understood). By the above-men- 
tioned measurement, position of the marker 1 26 with the world coordinate system 127 being as reference and position 
of the marker 126 with the coordinate system 129 being as reference can be recognized (understood). 
[0030] When the relationship between the world coordinate system 127 and the coordinate system 129 is assumed 

is to be the relationship in which the rotational transform R and the translations! transform T are implemented, the rela- 
tionship between position of the marker 126 with the world coordinate system 127 being as reference and position of 
the marker 126 with the coordinate system 129 being as reference also results in the relationship in which the rotational 
transform R and the trartslational transform T are implemented. Accordingly, since the position of the marker 126 with 
the world coordinate system 127 being as reference and position of the marker 126 with the coordinate system 129 

20 being as reference are known in advance, the rotational transform R and the translational transform T can be deter- 
mined by inverse operation. In practice, plural markers are used to determine rotational transform R and translational 
transform T by positions of plural markers. Accordingly, in what state (rotational transform R, translational transform T) 
current position and direction of the stereo camera are taken with the world coordinate system being as reference can 
be recognized (understood). 

25 [0031 ] Then, artificial background is determined by computer graphic. This artificial background is prepared with the 
world coordinate system being as reference. For example, white wall is assumed to be present on the surface of z=10. 
In practice, wall papers having pattern are prepared by CG to synthesize them. In this case, explanation will be given 
on the assumption that concerned wall paper is the simplest white wall. How the virtual white wall present at z=10 is 
viewed when image thereof is picked up (taken) from the virtual camera having positional relationship between R and 

30 T with respect to the origin of the world coordinate system is calculated. Picture when the virtual white wall is viewed 
from the virtual position is called "CG background picture". 

[0032] Moreover, "plate on which content of news. etc. is written" is prepared by CG. This plate is also prepared with 
the world coordinate system being as reference. For example, a plate of suitable size is assumed to be present at z=8 
(camera side relative to the wall). How this plate is viewed when its image is picked up (taken) from the virtual camera 
35 having positional relationship between R and T with respect to the origin of the world coordinate system is calculated. 
Picture when the virtual plate is viewed from this virtual position is called "CG flip picture". 

[0033] The portion of blue of the original picture imaged on the main camera is removed. Thus, only the background 
is removed and only projected image of performer is left This picture is assumed to be called "key extracted picture". 
In this case, the marker is also removed because it is blue system. 
40 [0034] In the "key extracted picture", picture of the previously described "C^ f Bp picture" is inserted imo the rerrwved 
portion. This is assumed to be called first synthetic picture. At the first synthetic picture, performer and projected image 
of the plate are present, and other portions still remain in removed (extracted) state. 

[0035] In the first synthetic picture, the previously described "CG background picture" is inserted into the portion 
which still remain removed. This is assumed to be called second synthetic picture. The second synthetic picture is a 
45 picture as if performer and "plate on which content of news. etc. is written" exist with white wail being as background. 
This second synthetic picture is picture ultimately obtained. 

[00361 Let now consider the case where performer carries out performance while pointing "plate on which content of 
news. etc. is written". Since there is no plate in practice, performer must carry out performance while pointing that plate 
as if it exists at that place. Accordingly, there was the problem that performer is difficult to carry out performance. 

so [0037] Explanation will now be given with reference to the attached drawings. FIGS. 8 A, 8B, 8C and 80 are views 
respectively showing positions of performer within the studio at times tl . t2. t3. t4. FIGS. 9A, 9B. 9C and 90 are respec- 
tive views of second synthetic picture (synthetic picture ultimately obtained) at times t1 , t2, t3, t4. FIGS. 10A. 10B, 10C. 
10D are respective views where position of performer within the studio and position of virtual plate prepared by CG are 
assumed to be viewed from the upper side at times t1. t2. t3. t4. In this case. t1 <t2 <t3 <t4 holds. 

55 [0038] Initially, standing position of performer will be described with reference to FIGS. 8A, 8B, 8C and 8D. In respec- 
tive figures, the stereo camera picks up image from the lower direction of the figure toward the upper direction thereof 
At time t1. as shown in FIG. 8A, performer 131 is present at the right end of the studio 130. Also at time t2. performer 
131 is present at the right end of the studio 130 as shown in FIG. 8B. At time t3, as shown in FIG. 8C, the performer 
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131 has been mowed to the central portion of the studio 130. At time t4. as shown in FK3. 8D. the performer 131 is 
present at the central portion of the studio 130 to stretch arm 132 to point air. 

[0039] The performer is caused to carry out performance in this way. and pictures ultimately obtained by picking up 
image of this performer by the stereo camera to synthesize CG with image from the main camera are shown in FIGS. 
5 9A. 9B. 9C and 9D. 

[0040] At time t1 . as shown in FIG. 9A. "CG background picture (white wall 1 34)" is synthesized with projected image 
1 33 of performer obtained from the main camera. Since patterned wall paper is made up by CG as background picture 
in practice also as previously described, the portion of 134 results in projected image of the patterned wall paper. In this 
case, that wall is caused to be white. In addition, since performer is present at the right end of the studio, projected 

10 image 133 is located at the right side in FIG. 9A At this time. "CG flip picture* is not yet synthesized. 

[0041 j At time t2. as shown in FIG. 9B. "CG flip picture 135* and "CG background picture (white wall 134)" are syn- 
thesized with picture obtained from the main camera This synthetic picture is picture ultimately obtained at the tune t2. 
CG flip picture 135 is projected image of plate on which the content of news prepared by virtual CG. etc. is written. 
[0042] At time t3. since performer is moved to the central portion as shown in FIG. 9C. projected image 133 of the 

is performer is positioned at the central portion even on the screen. This is picture ultimately obtained at the time t3. 

[0043] At time t4, as shown in FIG. 9D. since the performer stretches arm. projected image of arm 136 of the per- 
former appears even on the screen. This image is picture ultimately obtained at time t4. 

[0044] For viewer who looks at pictures of FIGS. 9A. 9B. 9C and 9D. it looks like that the performer is located at the 
right end at time t1. At time t2. it looks as if plate on which the content of news, etc. is written suddenly appears at the 
20 left side. At time t3. it looks like that the performer approaches that plate. At time t4. it looks like that the performer points 
that plate and makes explanation. In a manner as stated above, the viewer is caused to produce illusions as if performer 
makes explanation while looking at the plate which has suddenly appeared. 

[0045] Namely, the viewer is caused to produce illusions as if he looks at picture from the camera which has taken 
picture of the studio shown in FIGS. 10A. 10B. 10C. 10D. At time t1. as shown in FIG. 10A. it looks like that performer 
25 131 is present at the right end within the studio 130. At time t2. as shown in FIG. 10B. it looks like that the plate 137 
suddenly appears at the left side. At time t3, as shown in FIG. 10C. it looks like that performer 131 approaches the vicin- 
ity of the plate 137 within the studio 130. At time t4. as shown in FIG. 10D. it looks like that performer 131 points the 
plate 137 within the studio 130 in the state where arm 132 is raised. 

[0046] As seen from this example, the feature of the virtual studio or the virtual set is that viewer is permitted to pro- 
30 duce illusions as if material (object) which does not exist in reality is synthesized by using CG to thereby provide object 
existing in reality (e.g.. the above-described plate). 

[0047] Meanwhile, in the case where performer carries out performance while pointing "plate on which content of 
news. etc. is written" as descrfoed above, when consideration is made from standpoint of performer, he must carry out 
performance thinking that virtual plate is present within nothing or null space. Since the performer cannot understand 
35 where the plate is present, the performer is carrying out performance while checking picture obtained by synthesizing 
the ultimate CG by monitor within the studio and thinking that "plate should be located at this position". Accordingly, 
there was the problem that performance is very difficult to be carried out. 

[0048] Namely, hitherto, when the virtual studio or the virtual set etc. is used to synthesize CG with imaged picture 
to find it to the viewer, the performer must carry out performance while thinking as H "object to be synthesized later' is 
40 present Accordingly, there was the problem that performer is difficult to carry out performance. 

Disclosure of the Invention 

[0049] This invention has been made in view of the above-described problems, and its object is to provide a picture 
45 processing method, a picture processing apparatus, a picture processing panel, and a recording medium which permit 
performer to easily carry out performance. 

[0050] A picture processing method and a picture processing apparatus according to this invention are adapted to 
remove only the portion of specific cokx area from original picture to further synthesize other picture with picture in 
which only object located at three-dimensional position included in designated three-diinensional area of the three- 

so dimensional position of object projected onto the original picture is taken out to provide synthetic output. 

[0051) Moreover, a picture processing method and a picture processing apparatus according to this invention are 
adapted to remove only the portion of specific color area from an original picture to further synthesize other picture with 
picture obtained by taking out only object located at three-dimensional position included in the three-dimensional area 
adjustably (variably) designated of three-dimensional position of object projected onto the original picture. 

ss [0052] Further, a recording medium according to this invention is adapted so that there is recorded processing pro- 
cedure including a step of removing only the portion of specific color area from an original picture, a step of taking out 
only object located at three-dimensional position included in the designated three-dimensional area of three-dimen- 
sional position of object projected onto the original picture, and a step of synthesizing other picture with picture from 
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which only object located at three-dimensional position included in three-dimensional area is taken out. 
[0053] In accordance with this invention, e.g., director who manages performance adaptive ty designates the three- 
dimensional area with time by the world coordinate system to thereby take out only projected image imaged on the main 
camera corresponding to the inside of that area. Thus. e.g.. actual plate to which frame which is not blue is attached is 
5 prepared to designate the area so as not to first include that actual plate to synthesize GP flip picture so as to cover 
actual plate with reference to the frame thereof to designate, for a second time, the area so as to include the actual 
plate. Thus, even if performer approaches the actual plate, projected image of performer can be extracted. Accordingly, 
the performer can carry out performance while pointing actual plate. 

[0054] Further, in this invention, in extracting performer, etc. from picture, the portion of blue is first removed to further 
io also remove the area except for the three-dimensional area that director who manages performance has designated. 
Accordingly, such an approach is employed to exclude background in the designated three-dimensional area. Thus, it 
is not required that pattern is blue system. Conspicuous pattern which is not blue system is attached, thereby making 
it easy to recognize projected image of pattern from that picture when photographing is carried out by the camera. Fur- 
ther, pattern imaged on that picture is outside the designated three-dimensional area, and is removed in key extraction. 
is Accordingly, there is no problem. 

[0055] In addition, hitherto, performer could not prepare synthetic picture as if performer carries out describe opera- 
tion, by pen. on the plate prepared by CG. In this invention, performer carries out write operation by pen onto picture 
processing panel. Further, since character written by pen is left in key extraction, such character is left also in the syn- 
thetic picture ultimately obtained. Namely, it is possible to prepare synthetic picture as if performer carries out describe 
20 operation by pen onto the plate prepared by CG. This is based on the feature of this invention that performer writes 
rough copy onto actual plate by color of blue system in advance, and writes it by pen of conspicuous cbkx which is not 
blue system, thereby making it possible to write it while also precisely aligning position with respect to CG. 

Brief Description of the Drawings 

25 

[0056] 

FIG. 1 is an external perspective view of conventional calibration unit. 

FIG. 2 is a view showing the state where two cameras are slid by the calibration unit shown in FIG. 1 
so FIG. 3 is a view showing the state where flat plate is slid by the calibration unit shown in FIG. 1 . 

FIG. 4 is a view showing the state where point on a predetermined line is projected onto a predetermined position 
of screen. 

FIG. 5 is a view for explaining a method in which stereo camera system determines position on three-dimension of 
object by using calibration data obtained by the calibration unit shown in FIG. 1 . 
35 FIG. 6 is a view for explaining rotational transform and translational transform by movement of the stereo camera. 
FIG. 7 is a view for explaining the relationship between world coordinates and coordinates with camera being as 
reference. 

FIG. 8A is a view when position of performer at time t1 within conventional virtual set studio is viewed from the 
upper side. FIG. 8B is a view when position of performer at tune t2 therewithin is similarly viewed from the upper 

40 side. FIG. 8Ctsa view when position of performer at time t3 therewithin is viewed from the upper side, and FIG. 8D 
is a view when position of performer at time t4 therewithin is viewed from the upper side. 
FIGS. 9A, 9B. 9C. 9D are views respectively showing synthetic pictures ultimately obtained at times tl. t2. t3. t4. 
FIGS. 10A. 10B. 10C. 10D are respective views in which position of performer within the studio and position of vir- 
tual plate prepared by CG are viewed from the upper side at times t1, t2. t3, t4. 

45 FIG. 1 1 is a block diagram of a picture processing apparatus serving as picture processing method and picture 
processing apparatus according to this invention. 

FIG. 12 is a flowchart showing the detail of picture processing method that picture processing apparatus serving 
as the embodiment executes. 

FIG. 13 is a view snowing studio inside serving as portion where the picture processing unit executes the picture 
so processing method. 

FIG. 14 is a view in which the state of the studio inside at time T1 is viewed from upper side. 

FIG. 1 5A is a view showing original picture in the state of FIG. 1 4. FIG. 15B is a view showing key extracted picture 

in the same state, and FIG. 15C is a view showing second synthetic picture in the same state. 

FIG. 1 6 is a view showing the state of the studio inside at time T2 viewed from the upper side. 
55 FIG. 17A is a view showing original picture in the state of FIG. 16. FIG- 17B is a view showing key extracted picture 

in the same state, and FIG. 17C is a view showing second synthetic picture in the same state, 

FIG. 18 is a view showing the state of the studio inside at time T3 viewed from the upper side. 

FIG. 19A is a view snowing original picture in the state of FIG. 1 8. FIG. 19B is a view showing key extracted picture 
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in the same state, and FK3- 19C is a view showing second synthetic picture in the same state. 
FIG. 20 is a view showing the studio inside at time T4 viewed from the upper side- 

FIG. 21 A is a view showing original picture in the state of FIG. 20. FIG. 21 B is a view showing key extracted picture 
in the same state, and FIG. 21 C is a view snowing second synthetic picture in the same state. 

5 FIG. 22 is a view showing the state of the studio inside at time T5 viewed from the upper side. 

FIG. 23 A is a view showing original picture in the state of FIG. 22. FIG. 23 B is a view showing key extracted picture 

in the same state, and FIG. 23C ts a view showing second synthetic picture in the same state. 

FIG. 24 is a view when the state of the studio inside at time t6 is viewed from the upper side. 

FIG. 25A is a view showing original picture in the state of FIG. 24. FIG. 25B is a view showing key extracted picture 

w in the same state, and FIG. 25C is a view showing second synthetic picture in the same state. 
FIG. 26 is a view when the state of the studio inside at time T7 is viewed from the upper side. 
FIG. 27 A is a view showing original picture in the state of FIG. 26. FIG. 27 B is a view showing key extracted picture 
in the same state, and FIG. 27C is a view showing second synthetic picture in the same state. 

is Best Mode for Carrying Out the invention 

[0057] Explanation will now be described with reference to the attached drawings in connection with preferred embod- 
iment according to this invention. 

[0058] This embodiment is directed to a picture processing apparatus as shown in FIG. 1 1 . and comprises a stereo 

20 camera 1 composed of a main camera 2 and a sub camera 3. a picture data storage unit 4 for storing picture data from 
the main camera 2 and the sub camera 3 of this stereo camera 1 . a CG data storage unit 5 for storing computer graphic 
(CG) data therein, a marker position information storage unit 6 tor storing position information of marker viewed from 
the world coordinate system in advance, an arithmetic (calculating) unit 7 for carrying out a series of calculations which 
will be described later on the basis of information inputted from the input unit 9, and a control unit 8 for controlling the 

25 picture data storage unit 4, the CG data storage unit 5 and the marker position information storage unit 6. 

[0059] The control unit 8 includes recording medium therewithin and is adapted to store therein program for allowing 
the arithmetic unit 7 to carry out a series of calculations which will be described later The control unit 8 suitably reads 
out on the basis of information inputted from the control unit 9, data from respective ones of the picture data storage 
unit 4. the CG data storage unit 5 and the marker position information storage unit 6 to deliver them to the arithmetic 

30 unit 7 to allow the arithmetic unit 7 to carry out suitable calculation. 

[0060] Namely, the control unit 8 sequentially takes out. from the recording medium included therewithin. processing 
procedure including a step of removing only the portion of specific color area, a step of taking out only object located at 
three-dimensional position included in the designated three-dimensional area of the three-dimensional position of the 
object projected onto the original picture, and a step of synthesizing other picture with picture thus taken out, thus to 

35 allow the arithmetic unit 7 to carry out a series of operations. In this case, the above-mentioned three-dimensional area 
is adjustably (variably) designated with time. 

[0061 ] Further, the control unit 8 also carries out a control to output, from the output terminal, second synthetic picture 
which is ultimately synthesized picture obtained by calculation at the arithmetic unit 7. 

[0062] In this example, this picture processing apparatus allows the arithmetic unit 7 to carry out the picture process- 
40 ing method according to this invention in accordance with control at the control unit 8. This picture processing method 
is processing procedure shown in the flowchart of FIG. 12 which will be described later in detail, and is a picture 
processing method, in broad outline, to remove only the portion of specific color area from the original picture to further 
synthesize other picture with picture in which only object located at three-dimensional position included in designated 
three-dimensional area of three-dimensional position of object projected onto the original picture is taken out to output 
45 H. In this case, the flowchart shown in FIG. 12 is stored at the recording medium within the control unit 8 as described 
above. 

[0063] The operation for allowing the picture processing apparatus to carry out such a picture processing to execute 
processing procedure as shown in FIG. 12 to synthesize CG with projected image of performer 10 actually imaged 
within the studio of FIG. 13 win now be described. 

bo [0064] Initially, the state of the studio inside shown in FIG. 13 will be described. While the stereo camera 1 is labeled 
in such a manner that the number of cameras is one, main camera 2 and sub camera 3 are integrated as shown in FIG. 
1 1 . Within the studio, performer 10 is present within space defined or surrounded by a channel-shaped wall 1 1 of blue 
serving as background. Markers 12 1( 12 2 , 12 3 . 12^, 12 5 . 12g • • • of pattern such that conspicuous colors which are 
not blue system, e.g.. outside cirde is red and inside circle is yellow are attached to the wall 1 1 of blue. 

ss [0065] By carrying out photographing within such studio, projected images o* the manners 12 1t 12 2 . 1 23. 12^ 1 25. 12s 
* • * are conspicuous among those pictures. Accordingly, it is easy to recognize them Moreover, white six markers are 
provided in this example, it is a matter of course that the number of markers may be larger or smaller than that. When 
a large number of markers I2 1a 12 2 . 12 3 , 12 4 , 12 S 12$ • • • are provided, even if there is any marker which is blocked 
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(shielded) by the performer 10 and ts not therefore projected onto the stereo camera 1 . other markers can be seen from 
the stereo camera i. 

[0066] Hitherto, marker of blue system is attached to the background of blue. The reason why the marker is caused 
to be blue system is as follows. Since the blue portion is deleted in extracting performer, etc. from picture later at the 

5 time of so-called key extraction to thereby carry out key extraction, the marker can be also deleted together with the 
background. However, there was the problem that when marker of blue system on the background of blue is photo- 
graphed by the camera, it is difficult to recognize projected image of the marker from that picture. 
[0067] On the contrary, in the picture processing apparatus according to this invention, in extracting performer 10. etc. 
from picture within the studio shown in FIG. 13. the portion of blue is first removed. Further, the portion except for the 

io three-dimensional area W which will be described later that the director has designated at the input unit 9 is also 
removed. Accordingly, the director makes a designation so as not to include wall 1 1 of blue on which markers 12,. 12 2 . 
12 3 . 12 4 . 12 5 . 12e • • * areattached.therebypermittingthemarkers^!. 12 2 . 12 3 . 12 4 . 12 S 12 6 • * * to have conspic- 
uous color which is not blue system. 

[0068] An approach is employed to attach conspicuous markers 1 2, . 1 2 2 . 1 2 3 . 1 2 4 . 1 2 5 . 1 2 6 * • • which are not blue 
is system onto the wall 11 of blue to thereby facilitate recognition of projected images of markers from that picture when 
those markers are photographed by camera. Further, the markers 12,. 12 2 . 12 3 . 12 4 . 12s, 12e * * • imaged on these 
pictures are outside the three-dimensional area W designated by the director and are removed in key extraction. 
Accordingly, this is no problem. 

[0069] Positions of these markers 12,. 12 2 . 12 3 , 124. 12s. 12 6 • • • a r © measured with the world coordinate system 
20 1 3 being as reference in advance, and are stored as marker position information in the marker position information stor- 
age unit 6 shown in FIG. 1 1 . 

[0070] The operation explanation using the flowchart of FIG. 12 will now be carried out. There are also instances 
where respective steps of this flowchart may not be executed in dependency upon circumstances of the studio inside. 
[0071 ] Initially, the director who manages photographing within the studio designates three-dimensional area W at the 
25 world coordinate system 13 by input unit 9 at step S1 of FIG. 12. 

[0072] Thus, the control unit 8 reads out. from the picture data storage unit 4. picture data obtained from the main 
camera 2 and the sub camera 3 constituting the stereo camera 1 to deliver it to the arithmetic unit 7. The arithmetic unit 
7 calculates position on three-dimension (dimensional space) of the marker. 

[0073] Then, at step S3, the control unit 8 reads, from the marker position information storage unit 6, three-dimen- 
30 sional position in which the work! coordinate system 13 measured in advance is caused to be reference. At the arith- 
metic unit 7. in what state (rotational transform R. translational transform T) cunent position and direction of the stereo 
camera 1 are taken with the world coordinate system 13 being as reference is calculated from position on three-dimen- 
sion (dimensional space) of marker when viewed from the three-dimensional coordinate system 14. Since calculation 
for determining the rotational transform R and the translational transform T is carried out by using picture from the 
25 stereo camera 1 at respective times, those transforms can be precisely determined even in the case where the stereo 
camera 1 is moved. 

[0074] Since cokx of blue system is conventionally used in the marker, its recognition is very dm icutt. For example, 
when light irradiated onto the wall is changed, there were cases where there results the state where those markers can- 
not be recognized. However, in the studio where picture processing apparatus serving as this embodiment is used. 
40 markers 12,. 12 2 , 12 3 . 12 4 . 12 5 . 12e • • • which are not blue system are used. Accordingly, recognition is easy and it 
is possfole to precisely determine position and direction of camera. 

[0075] Then, at step S4. the control unit 8 sends, to the arithmetic unit 7. three-dimensional area W that the director 
has inputted from the input unit 9 at the step Si to allow the arithmetic unit 7 to calculate the three-dimensional area V 
when viewed from position and direction of the stereo camera 1 determined at the step S3. Namely, the arithmetic unit 
45 7 calculates, by control of the control unit 8. three-dimensional area V in which rotational transform R and translational 
transform T have been carried out with respect to the three-dimensional area W at the world coordinate system. 
[0076] Then, at step S5. the control unit 8 reads out picture data imaged by the main camera 2 of the stereo camera 
1 from the picture data storage unit 4 to send it to the arithmetic unit 7. The arithmetic unit 7 removes the portion of blue 
from this original picture. 

so [0077] Then, at step S6. the control unit 8 reads out picture data imaged by respective cameras 2 and 3 of the stereo 
camera 1 from the picture data storage unit 4 to send it to the arithmetic unit 7. The arithmetic unit 7 calculates parallax 
of the cameras 2 and 3 from these picture data to calculate position on three-dimension (dimensional space) of object 
(real thing) corresponding to all projected images. In this case, this position is determined as position in the three- 
dimensional coordinate system in which the stereo camera 1 is caused to be reference. 

55 [0078] Then, at step S7. the control unit 8 allows the arithmetic unit 7 to determine only object (real thing) within the 
three-dimensional area V determined at the step S4 from position on three-cSrnension of object (real thing) determined 
at the step S6. Further, the arithmetic unit 7 allows only projected image onto the main camera 2 of the object (actual 
thing) provided within three-dimensional area V to be left to remove other portion from the original picture. Thus, it is 
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possftae to prepare picture in which "biue portion" and the area "except for area V" have been aD deleted, so-called "key 
extracted picture" from original picture. 

[0079] Then, at step S8. the control unit 8 controls the arithmetic unit 7 to search pattern in which frame of red. etc. 
is registered in advance from picture data imaged by the main camera 2 and stored in the picture data storage unit 4. 
[0080] Then, at step S9. the control unit 8 allows the arithmetic unit 7 to synthesize "CG flip picture" with the portion 
deleted in the "key extracted picture" prepared at the step S7 to prepare first synthetic picture. The position where syn- 
thesis is earned out is the same position as the projected image position of pattern detected at the step S8. 
[0081] Then, at step S10. the control unit 8 allows the arithmetic unit 7 to synthesize "CG background picture" with 
the portion being continued to be still deleted even rf "CG flip picture" is synthesized within the first synthetic picture to 
prepare ultimately obtained second synthetic picture to be offered (provided) to viewer. 

[0082] Further, finally, at step S1 1 . the control unit 8 allows the arithmetic unit 7 to output second synthetic picture 
from the output terminal therefrom. 

[0083] The detailed operation in the case where the picture processing unit is caused to execute the processing pro- 
cedure of the flowchart shown in FIG. 12 within the studio shown in FIG. 13 will be described with reference to FIGS. 
14 to 27. 

[0084] Initially. FIG. 1 4 is a view showing that the state of the studio inside shown in FIG. 1 3 at time T1 is viewed from 
the upper side. Original picture imaged on the main camera 2 in the state of FIG. 14 is shown in FIG. ISA. Moreover, 
picture from which unnecessary portion has been removed of the original picture, i.e.. "key extracted picture" is shown 
in FIG. 15B. In addition, synthetic picture to be offered (provided) to the viewer ultimately obtained, i.e.. second syn- 
thetic picture is shown in FIG. 1 5C. 

[0085] Moreover. FIG. 1 6 is a view when the state of the studio inside at time T2 is viewed from the upper side. FIGS. 
17A, 17B. 17C respectively show original picture imaged on the main camera 2 in the state of FIG. 16. "key extracted 
picture" from which unnecessary portion has been removed of the original picture in that state, and second synthetic 
picture to be offered (provided) to viewer ultimately obtained in that state. In this case, the time T2 is time subsequent 
to time T1 . 

[0086] Further. FIG. 18 is a view when the state of the studio inside at time T3 is viewed from the upper side. FIGS. 
19 A, 19B. 19C respectively show original picture, "key extracted picture" and second synthetic picture in the state of 
FIG. 18. In this case, time T3 is time subsequent to time T2. 

[0087) Further. FIG. 20 is a view when the state of studio inside at time T4 is viewed from the upper side. FIGS. 21 A, 
21 B and 21 C respectively show original picture, "key extracted picture" and second synthetic picture in the state of FIG. 
20. In this case, time T4 is time subsequent to time T3. 

[0088] Further, FIG. 22 is a view when the state of the studio inside at time T5 is viewed from upper side. FIGS. 23A, 
23B and 23C respectively show original picture, "key extracted picture" and second synthetic picture in the state of FIG. 
22. In this case, time T5 is time subsequent to time T4. 

[0089] Further. FIG. 24 is a view when the state of the studio inside at time T6 is viewed from upper side. FIGS. 25A, 
25B and 25C respectively show original picture, "key extracted picture' and second synthetic picture in the state of FIG. 
24. In this case, time T6 is time subsequent to time T5. 

[0090] Further. FIG. 26 is a view when the state of the studio inside at time T7 is viewed from the upper side. FIGS. 
27A. 27B and 27C respectively show original picture, "key extracted picture" and second synthetic picture in the state 
of FIG. 26. In this case, time t7 is time subsequent to time T6. 

[0091] As shown in FIGS. 14. 16. 18. 20. 22. 24 and 26. stereo camera 1. performer 10 and walM 1 of blue serving 
as background in which marker is attached are disposed within the studio. In addition, although not shown in practice, 
three-dimensional area 15 that the director has designated is descrbed. This three-dimensional area 15 is caused to 
be area W in the world coordinate system. 

[0092] Director can designate three-dimensional area (W) 15 at respective times. In practice, such three-dimensional 
area (W) is selected from the pattern of the area in the world coordinate system registered in advance. Here, any one 
of patterns of two kinds of areas is selected. The threenlimensional area (W) shown in FIGS. 14. 16 and 26 is the first 
pattern, and the three-dimensional area (W) shown in FIGS. 1 8. 20. 22 and 24 is the second pattern. Namely, at times 
T1 . T2. T7. director designates the first pattern. At times T3. T4. T5. T6. director designates the second pattern. 
[0093] Moreover, in FIGS. 1 6. 1 8. 20. 22. 24 and 26. plate 20 is descrbed. This plate 20 is constituted by blue, and 
the portion that performer 10 writes by pen later in point of time is drafted (caused to undergo rough copy) by blue sys- 
tem Fringe is made on this plate 20. At this fringe portion, frame of conspicuous color, e.g.. red which is not blue system 
is attached. It is to be noted that "portion that performer writes by pen later" is the content that performer is desired to 
write on the spot at "plate on which content of news, etc is written" of CG to be synthesized, and is determined in 
advance. 

[0094] At time T2 in the state shown in FIG. 16. this plate 20 is carried through the outside of the three-dimensional 
area (W) is designated at that time. Further, for a time period from times T3 to T6, this plate 20 is placed within the 
studio as shown in the figure. At time T7 in the state shown in FIG. 26. this plate 20 is carried through the outside of 
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area IS designated al that time. Performer 10 is present at the right end of the studio at times T1 to T3. and is then 
moved to the side of plate 20. Accordingly, the performer 10 is present at the central portion at times T4 and T5. Further, 
sine© the performer 10 is then moved to the right end. it is at the right end at times T6 and T7. 

[0095] Respective A of FIGS. 15. 17. 19, 21. 23. 25 and 27 are original pictures imaged on the main camera 2 as 
s described above. CG is synthesized with original picture imaged on this main camera 2 to thereby prepare ultimate syn- 
thetic picture. Projected image 16 of performer is included in the original picture. Moreover, at this original picture, there 
is also projected image 17 of wall 1 1 of blue serving as background where marker is written. Generally, in the projected 
image 17 of the wall 11 of blue, plural projected images 18 of markers 12 v 12 2 . 12 3 . 12 4 , 125. 12e * * * shown in FIG. 
13 are included. In the figure, those projected images are collectively represented as one projected image 18. 
10 10096] In the original pictures of respective A of FIGS. 1 7. 21 . 23. 25 and 27. projected image 21 of plate 20 actually 
existing is imaged. The projected image 21 is composed of a projected image 21 , of red frame of plate 20. a projected 
image 21 2 of the portion of rough copy of the portion that the performer writes by pen later, and a projected image 21 3 
of the plate except therefor. The projected image 21 ^ of frame of plate is the portion of red. The projected image 21 2 of 
the rough copy portion is blue system. The portion 21 3 of the projected image of the plate except therefor is blue, 
is 10097] Unnecessary portion is removed from this original picture to prepare key extracted picture to further synthesize 
CG to thereby prepare final second synthetic picture. 

[0098) Respective B of FIGS. 15. 17. 19. 21. 23. 25 and 27 are picture in which unnecessary portion is removed of 
the original picture, i.e.. key extracted picture as described above. 

[0099] In these key extracted pictures, projected image 1 6 of performer existing within original picture at each time is 
so left as necessary portion in the state where it is not removed. In addition, the portion where the unnecessary portion is 
removed results in nothing or null portion 19. 

[0100] In respective B of FIGS. 19, 21 . 23 and 25. projected image 21 , of plate frame existing within original picture 
at each time ts also left as necessary portion in the state where it is not removed. 

[0101] Respective C of FIGS. 15. 17. 19. 21 , 23. 25 and 27 are second synthetic picture offered (provided) to the 
?s viewer finally obtained as described above. In this second synthetic picture, projected image 16 of performer 10 is left. 
In addition, projected image 21 , of plate frame is also left in respective C of FK3S. 19. 21 . 23 and 25. 
[0102] Moreover, while description is made in FIGS. 14. 16, 18. 20. 22. 24 and 26 such that stereo camera 1 is fixed, 
it may be moved in practice. As explained with reference to FIG. 13. it is possible to specify three-dimensional position 
of that object on the basis of positions of projected images imaged on respective cameras at respective times. In this 
30 case, the operation to specify such position is carried out in the coordinate system in which stereo camera 1 at that time 
is caused to be reference. At respective times, it is possible to determine position and direction of stereo camera 1 at 
every time from projected images of markers 12 v 12 2 . 12 3 . 12 4 . 12s. 12$ • • • on the wall 1 1 of blue serving as back- 
ground. 

[0103] The processing procedure of the flowchart of FIG. 1 2 will now be described in accordance with times T1 to T7. 
35 After the processing at a certain time is completed, the flowchart of FIG. 12 is repeated as processing at the next time. 
In this case, steps which are not processed can exist in dependency upon circumferences of the studio inside at times 
Tl to T7 as described above. 

[0104] At time T1 . three-dimensional area (W) 15 of FtG. 14 is selected by the director, and is registered as set of 
points on three-dimension in which the world coordinate system 13 shown in FIG. 13 is caused to be reference. As pre- 

40 vkxisly described, it is posstole to recognize in what state (rotational transform R. translational transform T) position 
and/or direction of the stereo camera 1 at time T1 take with respect to the world coordinate system 1 3. Accordingly, the 
same transform as above is carried out with respect to the area (W) 15 to assume this to be three-dimensional area (V). 
Namely.V = {point where rotational transform R and translational transform T are carried out with respect to respective 
points on the three-dimensionaJ area W) . Transform from W to V is nothing but an only operation to merely transform 

45 area (W) 15 of FIG. 14 viewed from the world coordinate system 13 into coordinate system 1 4 of FIG. 13 with stereo 
camera 1 at time Tl being as reference. 

[01 OS] Then, the portion of blue of original picture shown in FIG. 15A is removed. Thus, the portion except for marker 
of background of blue is removed from projected image 17. Since trree-dimensional positions of respective bodies 
within the original picture are measured at the coordinate system 14 with stereo camera 1 at time T1 being as refer- 
so ence, object in which its position is not included within the area V ts removed. Thus, projected image 18 of marker 
attached to the background is removed. The reason thereof is that this portion is not included in three^imenstonal area 
V. Accordingly, key extracted picture after unnecessary portion has been removed in this way is as shown in FIG 1 SB. 
Projected image 16 of performer is left in this work. In addition, since background 17 (including projected image 18 of 
marker) shown in FIG. 15A is deleted, this portion results in nothing or null portion 19 as shown in FIG. 15B. 
55 [0106] Since projected image of "plate on which content of news, etc is written" prepared by CG is not synthesized 
at this time point Tl . only projected image of "white wall" which is background prepared by CG is synthesized. "White 
wall" virtually exists at position of z=10 with. e.g.. the world coordinate system being as reference. 
[0107] Initially, in what state (rotational transform R, translational transform T) position and direction of the stereo cam- 
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era 1 at time T1 are taken with respect to the world coordinate system is examined. Further, how virtual white wail 
present at z=10 is viewed when imaged from vidua) camera having positional relationship between the rotational trans- 
form R and the tra relational transform T with respect to the origin of the world coordinate system is calculated. Picture 
when the virtual white wall is viewed from virtual position is called *CG background picture" at time T1 . 
5 [01 08] In the "key extracted picture" at time Tl . picture of *CG background picture" is inserted into the removed por- 
tion. This picture is second synthetic picture shown in FIG. 15C. As a result, the second synthetic picture is picture as 
rf performer exists in the state where the white wall is caused to be background. Namely, projected image 16 of per- 
former is imaged on second synthetic picture of FIG. 15C in the state where "white wall" 20 prepared by CG is caused 
to be background. 

w [0109] At time T2 shown in FIG. 16. the same three-dimensional area (W) 15 as that of FIG. 14 is selected by the 
director. Initially, three-dimensional area V is determined similarly to that at time Tl Namely, since in what state position 
and direction of the stereo camera 1 at time T2 are taken with respect to the world coordinate system can be recog- 
nized, the same transform as above is carried out with respect to the three-dimensional area ( W) 1 5. In this case, it is 
not limited that R and T at time T2 are the same as R and T at time T1 . The reason thereof is that the stereo camera 1 

is may be moved. Also at times subsequent thereto, there are also instances where R and T are changed. Accordingly, 
even rf the three-dimensional area W is the same, the three-dimensional area V changes with time. 
[01 1 0] Moreover, at time T2. actually existing plate 20 is carried through the passage except for the three-dimensional 
area (W) 15 shown in FIG. 16. Accordingly, in the original picture shown in FIG. 17A at time T2. projected image 21 of 
the actually existing plate 20 is caused to undergo frame-in. i.e.. falls within the original picture. 

20 [01 1 1 ] In the original picture shown in FIG. 1 7 A, the portion of blue (and blue system) is (are) removed. Further, object 
in which that position is not included within the three-dimensional area V is removed. Thus, background 1 7 of FIG. 1 7(A) 
(including projected image 18 of marker) and projected image 21 of plate including plate frame are deleted. Projected 
image 16 of performer is left in this work. After such removing work, key extracted picture shown in FIG. 17B can be 
obtained. 

25 [0112] Since projected image of "plate on which content of news. etc. is written" prepared by CG is not synthesized 
even at this time T2. only projected image of "white wall" serving as background prepared by CG is synthesized. As pre- 
viously described. VhKe wall" virtually exists at position of z= 10 with the world coordinate system being as reference. 
[0113] In the "key extracted picture" at time T2. picture of "CG background picture" at time T2 is inserted into the 
removed portion. This picture is second synthetic picture shown in FIG. 1 7C. The second synthetic picture results in 

30 picture as rf performer exists with white wall being as background. Namely, in the second synthetic picture, projected 
image 16 of performer is imaged on "white wall" 20 prepared by CG. 

[0114] At time T3, three-dimensional area (W) 15 as shown in FIG. 18 is selected by the director. Similarly to times 
until now, the same transform as position and direction (rotational transform R. translation a! transform T) of the stereo 
camera at time T3 with respect to the world coordinate system is carried out with respect to the three-dimensional area 

35 W to obtain three-dimensional area V. 

[01 1 5] In the original picture shown in FIG. 19A. the portion of blue (and blue system) is (are) removed. Further, object 
in which that position is not included within the V is ateo removed. Thus, background 17 (including marker 18 portion) 
shown in FK3. 19A and projected image of plate except for the frame are deleted. Further, projected image 16 of per- 
former and projected image 21 , of frame of plate are left in this work. The reason thereof is that since projected image 

40 1 7 of the portion in which there is no marker within the background and projected images 21 2 , 21 3 of plate except for 
frame 21, of red of the plate are blue (or blue system), they are removed. Since the projected image 18 of marker 
attached to the background is not included in the three-dimensional area V, they are removed. The figure after such 
removing work results in key extracted picture shown in FIG. 19B. 

[0116] At this time T3. synthesis of projected image of "plate on which content of news. etc. is written" prepared by 
45 CG is carried out To realize this, the portion of frame of red is searched. As search method, two methods are conceiv- 
able. In this case, either one of methods may be used. The first method is a method of searching red frame on three- 
dimension. Since the stereo camera is used, three-dimensional positions of respective bodies can be measured. In 
view of the above, it is sufficient to find out frame of red within the measured object In more practical sense, object 
which is. e.g.. red and line segment on three-dimension is selected. If the number of selected line segments is lour, they 
so are considered as the portion of the frame of red. If the number of line segments is three or less, other tine segment of 
red is predicted to allow the number of line segments to be tour to allow them to be frame of red. Such phenomenon 
takes place in the case where a portion of red frame cannot be seen because performer, etc is present at the stereo 
camera side relative to red frame. If three-dimensional position of the portion of frame of red can be recognized, position 
of its projected image 21 , in the original picture of FIG. 19A can be recognized. Accordingly, also in the "key extracted 
55 picture" shown in FIG. 19B, position of projected image 21, of red frame can be seen. Even if a portion of the red frame 
is not imaged because H does not overlap with performer, where projected image of red frame exists in the "key 
extracted picture" can be expected. 

[011 7] Another method is a method of directty searching projected image of red frame on the two-dimensional plane 
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in the "key extracted picture" shown in FIG. 19B. In more practical sense, object which is, e.g., red and fine segment on 
the tvwo-dimension is selected, If the number of selected line segments is four, those line segments are portion erf pro- 
jected image of frame of red. If the number of line segments is three or less, other line segment of red is predicted to 
allow the number of line segments to be lour to allow them to be projected image of frame of red. Even if any method 

5 is used, it is possible to determine projected image of red frame in the "key extracted picture". 

[01 18] Then, projection transform is implemented so as to just fall within frame 21 ^ of red within the "key extracted 
picture" with respect to "plate on which content of news. etc. is written" prepared by CG. The projected image (CG flip 
picture) of the plate of CG to which this projection transform is implemented is synthesized with "key extracted picture". 
Position where projected image of plate of CG is synthesized is the removed portion in the "key extracted picture". In 

to other words. "CG flip picture" is first prepared to overwrite "key extracted picture" thereon. Thus, the projected image 
left of the projected image of plate of CG is only the projected image positioned at the nothing or null portion 19 within 
"key extracted picture". Namely, there is no possibility that projected image 16 of performer and projected image 21 1 of 
red frame are overwritten by "CG flip picture". In this way. "CG flip picture" is synthesized with "key extracted picture". 
This synthetic picture is called first synthetic picture at time T3. 

15 [01 1 9] Also in the first synthetic picture obtained by synthesizing "CG flip picture" with "key extracted picture", portion 
which is still nothing or null exists. Then, projected image of "white wall" which is the background prepared by CG is 
synthesized with this portion. As previously described, "white wall" virtually exists at position of z=10 with the world 
coordinate system being as reference. 

[0120] In the first synthetic picture at time T3. picture 20 of "CG background picture" at time T3 is inserted into the 
20 nothing or null portion. This picture is called second synthetic picture. As shown in FIG. 19C. the second synthetic pic- 
ture is a picture as if performer and red frame attached "plate on which content of news, etc. is written" exist in the state 
where white wall is caused to be background. Namely, projected image 16 of performer is imaged on "white wall" 20 at 
the second synthetic picture along with "plate on which content (A) of news. etc. is written" present at the inside 22 of 
the projected image 21 ^ of the red frame. 
25 [0121] At time T4 shown in FIG. 20. the same three-dimensional area (W) 15 as that of FIG. 18 is selected by the 
director. Similarly to that at times until now. the same transform as that of position and direction (rotational transform R. 
translational transform T) of the stereo camera 1 at time T4 with respect to the world coordinate system is carried out 
with respect to the three-dimensional area W to obtain three-dimensional area V. 

[01 22] Moreover, at time t4. the performer 1 0 approaches plate 20. In this case, since the plate 20 is actually existing 
30 plate, it is easy that the performer approaches that plate. Hitherto, since virtual plate is present at the portion where no 
object exists and performer approaches it. performer was very difficult to carry out performance. 
[0123] In the original picture shown in FIG. 21 A at time t4. since performer 10 approaches plate 20 in FIG. 20. pro- 
jected image 16 of performer partially overlaps with projected image 21 , of plate frame. Since performer is present in 
front of the plate, a portion of projected image 21 , of plate frame is not imaged. 
35 [01 24] In the original picture shown in FIG. 21 A, the portion of blue {and blue system) is (are) removed. Further, object 
in which that position is not included within the three-dimensional area V is removed. Thus, background 1 7 (including 
marker portion) shown in FIG. 21 A and projected image 21 of plate except for frame are deleted. Further, projected 
image 16 of performer and projected image 2^ of frame of plate are left in this work. The picture after such removing 
work is "key extracted picture" shown in FIG. 21 B. 
40 [01 25] Even at this time, synthesis of projected image of "plate on which content of news, etc. is written" prepared by 
CG is carried out To realize this, the portion of frame of red is searched by using the same method as the previously 
described method. As seen from the original picture shown in FIG. 21 A, one side of red frame is concealed by per- 
former. However, its one side can be estimated from remaining three sides. 

[01 26] Then, projection transform is implemented so as to just fall within the frame of red within "key extracted picture" 
45 with respect to "plate on which content of news, etc. is written" prepared by CG. The projected image of plate of CG 
(CG flip picture) to which such projection transform has been implemented is synthesized with "key extracted picture". 
The position where projected image of plate of the CG is synthesized is only the removed portion in the "key extracted 
picture". Namely, there is no possibility that projected image 16 of performer and projected image 21 A of red frame are 
overwritten by "CG flip picture". This synthetic picture is called first synthetic picture at time T4. 
so [0127] Also in the first synthetic picture in which "CG flip picture" is synthesized with "key extracted picture", portion 
which is still nothing or null exists. Then, projected image of "white wall" which is background prepared by CG is syn- 
thesized with this portion. The projected image of "white wall" to be synthesized (CG background picture) is picture 
when viewed from virtual position similarly to that at times until now, and its explanation is omitted. The picture into 
which picture oTCG background picture" is inserted into the nothing or null portion in the first synthetic picture at time 
ss T4 is second synthetic picture shown in FIG. 21 . This picture results in a picture as if performer and red frame attached 
"plate on which content of news, etc. is written" exist with the white wall being as background In addition, performer is 
such a picture present in the vicinity of the plate. 

[0128] At time T5 shown in FIG. 22. the same three-dimensional area (W) 1 5 as that of FIG. 20 is selected. Similarly 
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to times until now. the same transform as that of the position and the direction (rotational transform R. translationai 
transform T) of the stereo camera 1 at time T5 with respect to the world coordinate is carried out with respect to the 
area W to obtain three-dimensional area V. 

[01 29] Moreover, at the time T5. the performer 10 protrudes arm 23 as shown in FIG. 22 to describe graphic by pen 
5 toward the plate 20. In this case, the content to be written is drafted (caused to undergo rough copy) by blue system on 
the plate 20. Namely, since it is sufficient that the performer 10 carries out describe operation by pen along the rough 
copy, it is possfcle to easily describe the content. As color of characteristic or graphic written by pen. color which is not 
blue system is used. 

[0130] Since the performer 10 writes graphic by pen on the plate 20. projected image 24 of arm of performer exists 
w within original picture shown in FIG. 23 A at time T5. Further, since the rough copy portion 21 2 is written by the pen. it 
disappears, and projected image 25 of graphic written by pen can be seen in place thereof. In this connection, the rough 
copy portion 2 1 2 and projected image 25 of graphic are the same shape. The reason thereof is that the performer traces 
the rough copy portion by pen. 

[01 31 ] In the original picture shown in FIG. 23A. the portion of blue (and blue system) is (are) removed. Further, object 
is in which its position is not included within the three-dimensional area V is removed. Thus, background shown in FIG. 

23 A (including marker portion) and plate except for frame are deleted. Further, projected images 16. 24 of performer 

(including arm), projected image 21 , of frame of plate and projected image 25 of graphic written by pen which is not 

blue are left in this work. The picture after removing work is "key extracted picture" shown in FIG. 23B. 

[0132] Even at this time point, synthesis of projected image of "plate on which content of news. etc. is written" pre- 
20 pared by CG is carried out. To realize this, the portion of the frame of red is searched by using the same method as the 

previously described method. As seen from the original picture shown in FIG. 23 A. one side of red frame is concealed 

by performer. However, that one side can be estimated from other three sides. 

[0133] Then, projection transform is implemented to "plate on which content of news, etc. is written" prepared by CG 
so as to just fall within the red frame within the "key extracted picture". The projected image of plate of the CG (CG flip 

25 picture) to which this projection transform has been implemented is synthesized with "key extracted picture". The posi- 
tion where projected image of plate of the CG is synthesized is only the removed portion in the "key extracted picture". 
Namely, there is no possibility that projected images 16 and 24 of performer (including arm), projected image 21 of red 
frame and projected image 25 of graphic written by pen are overwritten by the "CG flip picture". This synthetic picture 
is called first synthetic picture at time T5. 

30 [01 34] Also in the first synthetic picture obtained by synthesizing "CG flip picture" with "key extracted picture*, portion 
which is not still nothing or null exists. Then, projected image of "white wall" which is background prepared by CG is 
synthesized with this portion. Projected image of "white wall" to be synthesized (CG background picture) 20 is picture 
viewed from virtual position similarly to that at times until now and its explanation is omitted. The picture into which pic- 
ture 20 of "CG background picture" is inserted into the nothing or null portion in the first synthetic picture at time T5 is 

35 second synthetic picture. This picture is a picture as if performer and red frame attached "plate on which content of 
news. etc. is written" exist in the state where the white wall is caused to be background. Further, this picture is picture 
in which graphic that performer writes is written. In this case, O * written by pen so as to surround character of A on 
the plate of CG. 

[01 35] At time T6. performer is also moved to the right end. Since the explanation at this time is substantially the same 
40 as explanation at time T3. its explanation is omitted, ft is to be noted that the point different from T3 is that performer 1 0 
writes graphic by pen on the actual plate 20. Accordingly, in the original picture at time T6 shown in FIG. 25A. there is 
graphic 25 in place of rough copy portion 21 2 in FIG. 1 9A. In addition, since this graphic 25 is not removed in key extrac- 
tion, it appears also at key extracted picture shown in FIG. 258 and second synthetic picture shown in FIG. 25C. 
[0136] The second synthetic picture shown in FIG. 25C results in a picture as if performer and red frame attached 
45 "plate on which content of news. etc. is written" exist with white wall being as background. Further, this picture is picture 
in which graphic 25 that performer writes is written on that plate. 

(01 37] Since the explanation at time T7 is substantially the same as the explanation at time T2. its explanation is omit- 
ted. It ts to be noted that the point different from time T2 is that performer writes graphic by pen on actual plate 20. How- 
ever, since the plate 20 is not included within the three-dimensional area V. projected image 25 of graphic is removed 

so along with projected image 2^ of red frame of plate in key extraction. Moreover, actually existing plate 20 is carried 
through the passage except for area (W) 15 shown in FIG. 26. Accordingly, in the original picture at time T7 shown in 
FIG. 27A. projected image 21 of plate is caused to undergo frame out in such a manner that the projected image 21 is 
gone out toward the outside of picture. In the original picture at time T7 shown in FIG. 27A, there is graphic 25 in place 
of rough copy portion 212 shown in FIG. 2 7 A. In addition, since graphic 25 is removed in key extraction, it does not 

55 appear on key extracted picture shown in FIG. 27B. and second synthetic picture shown in FIG. 27C. 

[0138] The second synthetic picture shown in FIG. 27C is a picture as if performer exists with white wall being as 
background. 

[0139] Further, although illustration is omitted, new plate is carried similarly to that at time T2 after that. Thus, per- 
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former can write graphic or character by pen onto new plate. In addition, it is possible to prepare synthetic picture as rf 
graphic or character is written on *ptate on which content of news. etc. is written" prepared by CQ 
[0140] As stated above, in the picture processing apparatus of this embodiment, since the portion of blue is removed 
and area except tor three-dimensional area that the director designates is ateo removed, marker of background can be 
5 removed in key extraction even if marker of blue system is not employed. Accordingly, cokx of marker is caused to be 
conspicuous color, whereby recognition of that projected image becomes easy when it is photographed by the cam- 
era. 

(0141 J Further, by the feature of this invention that the three-dimensional area that the director designates can be 
adjustably set with time, the following merits are provided. 

io [0142] At the process of caring-in or carrying-out of actual plate tor assisting play (performance) of performer, etc.. 
setting is made so as not to include that plate into the area to thereby remove the plate in key extraction. In such a state, 
when the performer approaches the plate, as far as performer is removed in key extraction. In this case, the director 
sets, for a second time, the area so as to include the plate before he approaches the plate, it is possible to avoid it. 
[0143] Further, the performer writes characters, etc. onto that plate, thereby making it possible to write it as if it is writ- 

is ten onto the ptate prepared by CG. 

[0144] While the background, etc. is caused to be blue and the blue portion is removed in key extraction in the above- 
mentioned explanation, it is a matter of course that color except for blue. e.g.. green, etc. may be employed. In addition, 
e.g.. color from which key is extracted may be caused to be red and the background may be caused to be red. 
[0145] Since the picture processing method and the picture processing apparatus according to this invention are 

20 adapted to take out only projected image imaged on the three-dimensional area designated by the world coordinate 
system by user, when the above-mentioned three-dimensional area is designated so as not to first include actual plate 
to which mark such as red frame is attached, even if other picture is synthesized so as to cover actual plate with refer- 
ence to the mark, the area is re-designated so as to include actual plate, thereby permitting performer to carry out play 
(performance) while designating actual plate. Thus, performance is facilitated. 

25 [0146] In addition, since the picture processing panel according to this invention is of the structure in which rough copy 
is carried out so that the character or the graphic is written by the same color as specific color of the background or color 
similar to specific color on the surface, play (performance) of performer can be facilitated 

[0147] It is to be noted that various modifications or applied examples are conceivable within the range which does 
not depart from the gist of this invention. Accordingly, the gist of this invention is not limited to the embodiment 

30 

Claims 

1 . A picture processing method for synthesizing at least two pictures, comprising: 
35 receiving an original picture from a main camera; 

removing only portion of a specific color area from the original picture; 

taking out only an object located at three-dimensional position included in a designated three-dimensional area 
40 of three-dimensional position of the object projected onto the original picture; and 

synthesizing other picture with the picture thus taken out to output it. 

2. A picture processing method as set forth in claim 1 . 

45 wherein the three-dimensional position of the object is determined by position on screen of projected image 

of the object imaged by plural imaging units. 

3. A picture processing method as set forth in daim 1 . 

wherein the specific cokx is blue or green. 

50 

4. A picture processing method as set forth in daim 1 . 

wherein the three-dimensional area is adjustably designated. 

5. A picture processing method as set forth in claim 4. 

55 wherein the three-dimensional area is adjustably designated with time 

6. A picture processing apparatus adapted for synthesizing at least two pictures, the apparatus comprising: 
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a receiving unit for receiving an original picture from a main camera; and 

an arithmetic unit for removing only portion of a specific color area from the original picture to further determine 
picture obtained by taking out only object located at three-dimensional position included within a designated 
three-dimensional area of the three-dimensional position of the object projected onto the original picture to 
synthesize other picture with this picture. 

7. A picture processing apparatus as set forth in claim 6. 

wherein the three-dimensional position of the object is determined by position on the screen of the projected 
image of the object imaged by plural imaging units. 

8. A picture processing apparatus as set forth in claim 6, 

wherein the specific color is blue or green. 

9. A picture processing apparatus as set forth in claim 6, 

wherein thg three-dimensional area is adjustably designated. 

1 0. A picture processing apparatus as set forth in claim 9, 

wherein the three-dimensional area is adjustably designated with time. 

1 1 . A picture processing panel used for picture processing adapted for removing portion of a specific color area from 
an original picture imaged by an imaging unit to further synthesize characters or graphics in the state where posi- 
tions on the three-dimension are caused to be in correspondence with each other to output them. 

wherein rough copy is implemented by color within color region removed from the original picture and visual 
color so that the characters or the graphics are written on the surface. 

1 2. A recording medium adapted so that there is recorded a processing procedure including 

a step of removing only portion of a specific color area from an original picture, 

a step of taking out only object located at three-dimensional position included within the designated three- 
dimensional area of the three-dimensional position of the object projected on the original picture, and 
a step of synthesizing other picture with picture in which only object located at three-dimensional position 
included within the three-dimensional area ts taken out. 



13. A recording medium as set forth in claim 12. 

wherein the three-dimensional area is adjustably designated. 
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